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NATIONAL  BUREAU  OF  STANDARDS 

The  National  Bureau  of  Standards1  was  established  by  an  act  of  Congress  March  3, 
1901.  The  Bureau's  overall  goal  is  to  strengthen  and  advance  the  Nation's  science  and 
technology  and  facilitate  their  effective  application  for  public  benefit.  To  this  end,  the 
Bureau  conducts  research  and  provides:  (1)  a  basis  for  the  Nation's  physical  measure- 
ment system,  (2)  scientific  and  technological  services  for  industry  and  government,  (3) 
a  technical  basis  for  equity  in  trade,  and  (4)  technical  services  to  promote  public  safety. 
The  Bureau  consists  of  the  Institute  for  Basic  Standards,  the  Institute  for  Materials 
Research,  the  Institute  for  Applied  Technology,  the  Center  for  Computer  Sciences  and 
Technology,  and  the  Office  for  Information  Programs. 

THE  INSTITUTE  FOR  BASIC  STANDARDS  provides  the  central  basis  within  the 
United  States  of  a  complete  and  consistent  system  of  physical  measurement;  coordinates 
that  system  with  measurement  systems  of  other  nations;  and  furnishes  essential  services 
leading  to  accurate  and  uniform  physical  measurements  throughout  the  Nation's  scien- 
tific community,  industry,  and  commerce.  The  Institute  consists  of  a  Center  for  Radia- 
tion Research,  an  Office  of  Measurement  Services  and  the  following  divisions: 

Applied  Mathematics  —  Electricity  —  Mechanics  —  Heat  —  Optical  Physics  — 
Linac  Radiation  -  —  Nuclear  Radiation  -  —  Applied  Radiation  -  —  Quantum 
Electronics  :i  —  Electromagnetics  '■  —  Time  and  Frequency  3  —  Laboratory 
Astrophysics  '  —  Cryogenics  3. 

THE  INSTITUTE  FOR  MATERIALS  RESEARCH  conducts  materials  research  lead- 
ing to  improved  methods  of  measurement,  standards,  and  data  on  the  properties  of 
well-characterized  materials  needed  by  industry,  commerce,  educational  institutions,  and 
Government;  provides  advisory  and  research  services  to  other  Government  agencies; 
and  develops,  produces,  and  distributes  standard  reference  materials.  The  Institute  con- 
sists of  the  Office  of  Standard  Reference  Materials  and  the  following  divisions: 

Analytical     Chemistry — Polymers — Metallurgy — Inorganic     Materials — Reactor 

Radiation — Physical  Chemistry. 

THE  INSTITUTE  FOR  APPLIED  TECHNOLOGY  provides  technical  services  to  pro- 
mote the  use  of  available  technology  and  to  facilitate  technological  innovation  in  indus- 
try and  Government;  cooperates  with  public  and  private  organizations  leading  to  the 
development  of  technological  standards  (including  mandatory  safety  standards),  codes 
and  methods  of  test;  and  provides  technical  advice  and  services  to  Government  agencies 
upon  request.  The  Institute  also  monitors  NBS  engineering  standards  activities  and 
provides  liaison  between  NBS  and  national  and  international  engineering  standards 
bodies.  The  Institute  consists  of  a  Center  for  Building  Technology  and  the  following 
divisions  and  offices: 

Engineering  and  Product  Standards — Weights  and  Measures — Invention  and 
Innovation — Product  Evaluation  Technology — Electronic  Technology — Techni- 
cal Analysis — Measurement  Engineering — Building  Standards  and  Code  Serv- 
ices4— Housing  Technology4 — Federal  Building  Technology4 — Structures,  Mate- 
rials and  Life  Safety4 — Building  Environment4 — Technical  Evaluation  and 
Application4 — Fire  Technology. 

THE  INSTITUTE  FOR  COMPUTER  SCIENCES  AND  TECHNOLOGY  conducts  re- 
search and  provides  technical  services  designed  to  aid  Government  agencies  in  improv- 
ing cost  effectiveness  in  the  conduct  of  their  programs  through  the  selection,  acquisition, 
and  effective  utilization  of  automatic  data  processing  equipment;  and  serves  as  the  prin- 
cipal focus  within  the  executive  branch  for  the  development  of  Federal  standards  for 
automatic  data  processing  equipment,  techniques,  and  computer  languages.  The  Center 
consists  of  the  following  offices  and  divisions: 

Information  Processing  Standards — Computer  Information — Computer  Services 
— Systems  Development — Information  Processing  Technology. 

THE  OFFICE  FOR  INFORMATION  PROGRAMS  promotes  optimum  dissemination 
and  accessibility  of  scientific  information  generated  within  NBS  and  other  agencies  of 
the  Federal  Government;  promotes  the  development  of  the  National  Standard  Reference 
Data  System  and  a  system  of  information  analysis  centers  dealing  with  the  broader 
aspects  of  the  National  Measurement  System;  provides  appropriate  services  to  ensure 
that  the  NBS  staff  has  optimum  accessibility  to  the  scientific  information  of  the  world, 
and  directs  the  public  information  activities  of  the  Bureau.  The  Office  consists  of  the 
following  organizational  units: 

Office  of  Standard  Reference  Data — Office  of  Technical  Information  and 
Publications — Library — Office   of   International   Relations. 

1  Headquarters  and  Laboratories  at  Gaithersburg,  Maryland,  unless  otherwise  noted  ;  mailing  address 
Washington,  D.C.  20234. 

•  Part  of  the  Center  for  Radiation  Research. 
3  Located  at  Boulder.  Colorado  80302. 

•  Part   of   the   Center   for   Building   Technology. 


!H%  u     - 

6    1973 

To    Project  SOAP:  A  Systems 
-        Approach  to  Biomedical  Research 
Program  Management  •  a  case  study 


Robert  S.  Cutler 


Technical  Analysis  Division 
Institute  for  Applied  Technology 
/fJ.J<   National  Bureau  of  Standards 
'"     Washington,  U.C.    20234 


NBS  Technical  Notes  are  designed  to  supplement  the 
Bureau's  regular  publications  program.    They  provide  a 
means  for  making  available  scientific  data  that  are  of 
transient  for  limited  interest.    Technical  Notes  may  be 
listed  or  referred  to  in  the  open  literature. 


/  w 


Ml 


*. 


U.S.  DEPARTMENT  OF  COMMERCE,  Frederick  B.  Dent,  Secretary 
NATIONAL  BUREAU   OF  STANDARDS,   Richard  W.  Roberts,  Director 


Issued  April  1973 


National  Bureau  of  Standards  Technical  Note  761 

Nat.  Bur.  Stand.  (U.S.),  Tech.  Note  761,  75  pages  (Apr.  1973) 
CODEN:  NBTNAE 


For  sale  by  the  Superintendent  of  Documents,  U.S.  Government  Printing  Office,  Washington,  D.C.  20402 
(Order  by  SD  Catalog  No.    013.46:761). 
Price  $0.95  domestic  postpaid  or  $0.70  G.P.O.  Bookstore 


Project  SOAP:   A  Systems  Approach  to 
Biomedical  Research  Program  Management 
A  Case  Study 

SYNOPSIS 

This  report  documents  a  case  study  in  the  field  of 
management  science  in  which  a  systems  approach  resulted  in 
the  implementation  of  a  discipline  for  managing  biomedical 
research  programs  within  a  Federal  research  support  agency. 
As  such,  it  has  value  for  practitioners  of  the  systems 
approach  in  itself,  and  also  as  a  potential  guide  for 
Science  Administrators  and  Budget  Analysts  elsewhere  in 
Government.   It  describes  a  method  for  allocating  funds  to 
basic/applied  research  activities,  particularly  the  planning 
and  evaluation  of  publicly "funded  project  grant  programs. 

Project  SOAP  (an  acronym  for  Systems/Operations  Analysis 
of  Programs)  was  the  name  given  a  joint  task  group  established 
under  an  interagency  agreement  between  the  Technical  Analysis 
Division  (TAD)  of  the  National  Bureau  of  Standards  (NBS)  and 
the  National  Institute  of  Allergy  and  Infectious  Diseases 
(NIAID)  of  the  National  Institutes  of  Health  (NIH) . 

This  case  study  describes  the  methodology  and  sub- 
stantive information  developed  during  Project  SOAP's  two- 
year  assignment  (from  July  1969  to  June  1971),  which 
resulted  in  the  introduction  of  a  discipline  for  program 
management  for  improving  the  administration  of  basic/applied 
research  project  grants  supported  by  NIAID.   Although  appli- 
cation of  the  quantitative  methods  of  operations  research  was 
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not  possible  during  this  effort,  the  basic  logic  of  the 
classical  resource  allocation  model  was  used  as  a  framework 
for  guiding  management  decisions. 

The  task  group  was  created  to  explore  the  applications 
of  systems  analysis  and  operations  research  to  management 
issues  within  the  National  Institute  of  Allergy  and 
Infectious  Diseases,  particularly  to  the  problem  of  resource 
allocation  in  its  Extramural  grants  program.   Together, 
the  task  group  investigated  the  methods  and  criteria  the 
Institute  used  for  selecting  projects  within  its  various 
research  programs. 

Initially,  it  was  found  that  the  Institute's  traditional 
administrative  practices  for  budget  allocation  did  not 
provide  an  adequate  basis  for  managerial  decision-making, 
since  attempts  to  establish  program  priorities  were 
ineffective.   The  agency  was  not  able  to  justify  to  higher 
authority  its  desired  choices  when  arbitrary  budget  cuts 
imposed  by  external  circumstances  seriously  impaired 
important  programs.   The  task  group  concluded  that  the  best 
solution  was  the  adoption  of  an  explicit  discipline  for 
management:   a  discipline  which  would  permit  more  effective 
control  over  what  had  become  a  major  cause  of  friction  and 
confusion  within  the  Federal-Science  environment.   This 
"dissonance"  stemmed  from  the  presence  of  two  value  systems: 
the  "scientific  merit"  criteria  (an  emphasis  of  the  Scienti- 
fic Community) ,  on  the  one  hand,  and  the  notions  of  public 


interest  (here  "health  relevance"  criteria,  as  conceived  by 
the  National  Advisory  Council),  on  the  other. 

The  benefits  which  accrue  to  this  Institute's  Science 
Administrators  from  the  full  implementation  of  such  an 
approach  include  not  only  greater  assurance  of  "optimum" 
resource  allocation  but  also  a  basic  pattern  of  reasoning 
useful  for  guiding  discussions  with  members  of  the 
Scientific  Community,  representatives  of  other  Federal 
health  research  agencies,  and  the  Office  of  Budget  and 
Management. 

Moreover,  it  has  become  evident  in  recent  years  that 
the  single  "peer  review"  procedure  for  selecting  grant 
payment  priorities  cannot  be  expected  to  be  responsive  to 
changing  National  priorities  unless  stronger  incentives  are 
provided  by  Science  Administrators  for  focusing  research 
investigators'  attention  on  nationally  important  health 
research  problems.   The  DAPA  (Dollar  Allocation  by  Program 
Area)  research  investment  procedure  recommended  by  the  task 
group  is  simply  a  more  deliberate  mechanism  through  which 
the  Science  Administrator  and  his  Advisory  Council  can  use 
the  advice  of  experts  from  the  Scientific  Community  jointly 
with  other  considerations  in  the  process  of  establishing 
grant  payment  priorities  for  the  Institute. 

The  author  favors  the  establishment  of  more  clear-cut 
and  explicit  planning  and  evaluation  procedures  for  the 
management  of  research  programs  within  the  Federal  Government. 
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Present  efforts  at  decentralizing  the  management  of  NIH's 
biomedical  research  activities,  with  their  emphasis  on 
Institute  determination  of  priorities  aimed  at  particular 
National  health  problems,  are  a  start  in  this  direction. 

The  author  also  believes  the  approach  described  in 
this  report  can  be  used  to  provide  Science  Administrators 
elsewhere  in  Government  with  an  approach  for  explicitly 
planning  for  and  evaluating  the  performance  of  their 
basic/applied  research  programs,  in  order  to  balance  the 
dollar  costs  with  the  expected  benefits  of  particular 
research  project  proposals. 

Within  the  National  Institute  of  Allergy  and  Infectious 
Diseases,  the  Extramural  Program  science  administrators  are 
currently  in  the  process  of  adopting  the  DAPA  procedure  to 
perform  this  function. 
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Project  SOAP:   A  Systems  Approach  to 
Biomedical  Research  Program  Management 
(A  Case  Study) 

Robert  S.  Cutler 


This  case  study  describes  the  activities  of  an  inter- 
agency task  group  that  applied  systems  analysis  to 
improve  management  controls  within  a  biomedical 
research  agency  of  the  federal  government.   The 
results  were  the  formulation  and  implementation  of 
a  discipline  for  program  management  which  explicitly 
makes  use  of  multiple  criteria  in  arriving  at 
resource  allocation  decisions. 

The  text  details  the  necessary  preliminary  analysis 
describing  operational  activities,  information  flows, 
and  key  decision  points  within  the  organization.   It 
goes  on  to  identify  the  techniques  employed  and  the 
difficulties  encountered  while  attempting  to  improve 
the  decision-making  process  for  selecting  research 
projects,  under  conditions  of  reduced  funding.   In 
particular,  a  comparison  is  made  between:   (1)  the 
agency's  traditional  single-criterion  "peer  review" 
judgment  for  determining  budget  priorities,  and 
(2)  the  multiple-criteria  judgments  required  to 
effect  more  positive  management  control.   The  sys- 
tematic use  of  separate  "scientific  merit"  and 
"health  relevance"  ratings  is  compared  with  the 
organization's  actual  experience.   The  criteria 
used  by  various  participants  in  the  decision  pro- 
cess are  analyzed,  and  a  dollar  allocation  "invest- 
ment" procedure  based  on  these  findings  is  developed. 

A  procedure  which  organizes  relevant  information  for 
research  program  planning  and  evaluation  is  presented, 
and  extension  of  this  recommended  procedure  to  wider 
use  by  science  administrators  elsewhere  in  govern- 
ment is  discussed. 

Keywords:   Biomedical  research;  "health  relevance"; 
program  planning  and  budgeting;  R§D  management; 
systems  analysis;  systems  approach. 

1.   INTRODUCTION 

The  National  Institute  of  Allergy  and  Infectious 

Diseases  (NIAID)  is  one  of  the  ten  National  Institutes  of 
Health.   It  supports  a  broad  range  of  biomedical  research  on 
the  causes  of  allergy  and  the  prevention  and  treatment  of 
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diseases  associated  with  such  infectious  agents  as  bacteria, 
viruses,  fungi,  and  parasites. 

The  budget  for  the  National  Institute  of  Allergy  and 
Infectious  Diseases  in  recent  years  has  been  approximately 
$100  million  a  year.   Half  of  this--some  $50  million--is 
appropriated  by  the  Congress  for  Extramural  research  and 
training  grants  which  support  biomedical  research  investiga- 
tors at  academic  institutions  throughout  the  United  States 
and  in  some  foreign  countries.   Of  the  approximately  1,700 
grant  applications  received  each  year  by  NIAID,  nearly  one- 
half  are  non-competitive  continuations;  i.e.,  they  are  long- 
term  commitments  made  by  the  Institute  in  previous  years  and 
are  based  on  prior  review  and  National  Advisory  Council* 
approvals . 

Of  the  remaining  800  to  900  applications,  about 
70  percent  are  judged  by  a  peer  review  procedure  to  have 
sufficient  scientific  merit.   However,  after  the  prior  year's 
commitments  have  been  paid  from  the  allotted  $50M,  only  some 


*The  National  Allergy  and  Infectious  Diseases  Council  consists 
of  a  group  of  13  to  15  prominent  biomedical  scientists, 
public  health  officials,  and  laymen  appointed  by  the 
Secretary  of  the  Department  of  Health,  Education  and  Welfare 
for  a  term  of  four  years.   This  Council  was  created  to  advise 
the  Director  of  NIH  on  policy  matters.   Moreover,  it  has  the 
statutory  responsibility  of  approving  recommendations  for 
research  project  grants  (through  the  Director  of  NIH)  to  the 
Surgeon  General  of  the  United  States.   Only  with  the 
Council's  express  approval  can  awards  for  Extramural  project 
grants  be  paid  by  the  Institute.   Formal  meetings  of  the 
Advisory  Council  are  scheduled  three  times  during  each  fiscal 
year,  in  November,  March  and  June.   Those  Extramural  grant 
applications  initially  accepted  by  a  peer  review  group  and 
favorably  evaluated  by  the  Institute  Staff  and  the  appropri- 
ate Council  subcommittee  are  finally  recommended  for  payment. 


two-thirds  of  those  remaining  grant  applications  recommended 

for  award  can  be  paid. 

As  early  as  1963,  when  the  previous  large  increases  in 
NIH  appropriations  first  began  to  level  off,  there  was  much 
discussion  among  the  NIAID  Staff  and  Advisory  Council  members 
about  the  desirability  of  giving  special  priority  to  certain 
areas  of  research  for  support  by  Extramural  Grants.   It  was 
recognized  that  such  areas  of  research  needed  more  support 
than  other  fields  because  they  dealt  with  problems  of  partic- 
ular importance,  or  held  promise,  or  had  been  neglected  in 

the  past. 

In  1964,  when  the  NIAID  Extramural  Grants  budget  was 

actually  restricted,  a  new  and  serious  problem  confronted 
the  Institute:   how  to  assure  the  means  for  support  for 
those  research  activities  which  the  Institute  considered 
most  relevant  to  its  categorical  mission  of  improving  public 
health  through  the  support  of  basic  science  and  the  perform- 
ance of  targeted  research.   The  problem  was  not  so  much 
deciding  how  to  determine  the  most  scientifically  meritorious 
projects  from  among  those  screened  by  the  initial  review 
groups  (Study  Sections)  and  referred  to  NIAID  for  financial 
support  but,  rather,  deciding  how  to  determine  the  preferred 
order  of  payment  of  those  research  project  proposals 
approved- -particularly  as  research  budgets  declined- -and  how 
to  make  this  kind  of  decision  on  a  more  rational  and  system- 
atic basis. 


Finally,  in  1967,  faced  with  the  practical  problem  of 
preserving  support  for  important  biomedical  research  projects 
while  at  the  same  time  faced  with  an  actual  cut  in  budget  for 
Extramural  Grants,  NIAID  responded  by  designating,  from  among 
the  many  possible  biomedical  research  areas,  ten  Special 
Emphasis  Research  Programs*  (SERPs)  as  the  ones  to  receive 
priority.   Grants  associated  with  these  areas  would  be  given 
a  "two-decile"**  advantage  in  payment  priority  over  grants 
associated  with  other  areas  of  research. 

This  procedure  became  the  subject  of  much  discussion 
within  the  NIAID  management.   It  was  called  "decile-diddling" 
by  some,  and  there  were  increasing  demands  for  justifying 
the  allocations  resulting  from  such  informal  procedures. 
In  essence,  the  rationality  of  the  allocation  process 
itself  became  an  important  issue. 

The  result  of  all  this  was  that,  in  1968,  NIAID 
agreed  to  take  a  more  sophisticated  look  at  various 
methods  for  determining  grant  payment  priorities  in  the 


*The  initial  NIAID  Special  Emphasis  Research  Programs  were: 
Drug  Resistance  and  Microbial  Diseases;  Streptococcal 
Infections  and  Sequelae;  Congenital  Defects  caused  by 
Microbial  Agents;  Antiviral  Substances;  Chronic  and 
Degenerative  Diseases;  Infectious  Hepatitis;  Emphysema 
and  Chronic  Lung  Disease;  Transplantation  Immunology; 
Clinical  Allergy  and  Immunology;  and  Malaria.   NIAID  Staff 
Paper  (June  5,  1967)  . 

**For  explanation,  see  pp.  22,  27-29. 


Extramural  program,  a  decision  largely  the  result  of  the 
appointment  of  Dr.  C.  West  Churchman*  as  a  lay  member  to 
the  NIAID  Advisory  Council. 

1 . 1   The  Necessity  for  Change 

When  the  change  in  Federal  budget  priority  for  supporting 
NIH  Extramural  research  resulted  in  the  uniform  cutbacks  of 
1969,**  it  became  obvious  to  many  within  the  Institute  that 
NIAID' s  external  environment  was  changing  and  that  perhaps 
their  basic  decision  criteria  ought  to  be  examined  in  a 
different  light. 

At  the  same  time,  after  several  Advisory  Council  meetings 
it  became  clear  that  different  interpretations  were  being  made 
by  different  Council  members  of  the  Study  Sections'  (peer 
review)  ratings  of  grant  requests.   That  this  ambiguity 
resulted  in  the  difficulty  when  these  priority  scores  were 
considered  by  the  Council  was  apparent,  in  particular,  from 
the  way  final  payment  priorities  for  certain  grants  differed 
from  those  intended  in  ratings  of  the  Study  Sections.   Some 
members  of  the  Advisory  Council  felt  that  the  use  of  Study 
Sections'  priority  scores  together  with  the  decile  adjustments 


*Dr.  C.  West  Churchman,  Professor  of  Business  Administration, 
University  of  California,  Berkeley,  and  a  leading  exponent 
of  the  "systems  approach." 

**In  July  1969,  NIH  instituted  an  across-the-board  20  percent 
cut  in  research  grants  up  for  renewal.   Later,  because  of 
vociferous  reaction  by  the  biomedical  research  community, 
DHEW  reduced  the  budget  cuts  to  5-10  percent. 


recommended  by  the  Institute  Staff  did  not  result  in  their 
notion  of  a  balanced  research  program,  or  at  least  did  not 
result  in  a  program  the  rationality  of  which  was  evident. 

In  seeking  a  change  for  the  better,  the  Institute  began 
to  address  itself  to  developing  more  explicit  goals  and 
criteria  for  evaluating  the  performance  of  its  programs. 
The  existing  ones  were  found  to  be  too  vague  and  inappro- 
priate for  planning  and  guiding  its  decision-making  process 
in  relation  to  its  public  responsibility  of  improving  the 
nation's  health  through  research. 

Dr.  Dorland  J.  Davis,  the  Director  of  NIAID,  best 
summed  up  the  problem  when  he  asked:   "What  actions  should 
the  Institute  take  to  assure  the  most  effective  and 
responsible  use  of  these  funds  entrusted  to  its  steward- 
ship?"  Further,  he  posed,  "How  can  we  be  assured  five 
years  from  now--with  present  budget  allocations- -that  some 
of  our  best  [research]  programs  won't  be  starving,  and 
our  worst  programs  flourishing?   Our  goal  ought  to  be  to 
improve  the  management  of  our  programs ,  not  to  perpetuate 
the  controls  we  now  exercise."* 

In  an  attempt  to  provide  this  desired  improvement, 
Project  SOAP  was  thus  established  as  a  joint  task  group 
under  an  interagency  agreement  between  the  Technical 
Analysis  Division  (TAD)  of  the  National  Bureau  of 


*Initial  meeting  with  Project  SOAP,  August  8,  1969. 
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Standards  (NBS)  and  the  National  Institute  of  Allergy 

and  Infectious  Diseases  (NIAID)  of  the  National  Institutes 

of  Health  (NIH) . 

The  objective  of  SOAP  (an  acronym  for  Systems/Operations 
Analysis  of  Programs)  was  to  explore  the  application  of 
systems  analysis  and  operations  research  to  significant 
management  problems  of  NIAID,  particularly  to  those  of  its 
Extramural  Research  Grants  Program. 

The  title  of  the  report  prepared  for  the  sponsor 
at  the  conclusion  of  the  assignment,  "Toward  a  Discipline 
for  Program  Management,"*  epitomized  the  basic  discipline 
used  by  the  Project  SOAP  task  group.   This  discipline 
introduced  to  research  scientists  and  science  administrators 
at  NIAID  a  more  consistent  and  rationally-based  framework 
for  resource  allocation.   The  "Program  Management  Discipline" 
relates  operational  decision-making  information  to  higher- 
level  considerations  of  health  research  and  public  policy. 
It  is  strategic  in  nature;  it  attempts  continually  to 
reexamine  goals  as  well  as  to  determine  the  appropriate 
means  for  achieving  the  desired  degree  of  managerial  control. 

Today,  in  practically  every  agency  of  the  Federal 
Government  there  are  operations  researchers,  management 
scientists,  system  scientists- -all  attempting  to  look  at 


*Cutler,  R.  S.  and  Martino,  V.  A.,  "Toward  a  Discipline  for 
Program  Management"  (A  Summary  of  Project  SOAP  Activities, 
FY  70-71),  National  Bureau  of  Standards  Report  No.  10-626 
(January  31,  1972) . 


the  problems  of  Government  from  the  so-called  "systems 
viewpoint."   As  scientists  by  profession,  they  are 
interested  in  characterizing  the  nature  of  operational 
systems  in  such  a  way  that  the  decision-making  function  can 
be  performed  in  a  logical  and  coherent  fashion.   Furthermore, 
by  applying  scientific  discipline  and  substantive  knowledge 
to  a  particular  field  of  activity,  they  expect  to  be  able 
to  develop  measures  which  will  give  as  adequate  information 
as  possible  about  the  performance  of  these  systems. 

This  report  documents  one  such  systems  analysis  study. 
No  claim  for  a  final  successful  solution  is  made.   Rather, 
the  following  text  describes  both  some  specific  "subsystem" 
analyses  and  the  steps  taken  toward  delineating  a  management 
decision-making  structure  within  which  the  Institute  itself 
later  began  to  undertake  more  comprehensive  and  definitive 
programs.   An  abstract  of  the  Official  Minutes  of  the 
June  1971  Council  meeting  describing  the  "first  step  in 
implementation  of  the  Dollar  Allocation  Procedure"  is  shown 
in  the  Conclusion  on  page  55. 

1 . 2   Task  Group's  Approach 

The  Institute's  interest  in  this  systems  analysis  pro- 
ject was  centered  in  the  top  echelon  of  the  organization. 
The  Institute  Director  assigned  two  men*  from  his  staff  to 


*Dr.  Alfred  M.  Webb,  Chief,  Office  of  Program  Planning  and 
Projections  (NIAID);  and  Charles  Myers,  Management  Analyst 
(NIAID),  later  reassigned. 
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join  on  a  half-time  basis  the  two  full-time  systems  analysts 
from  TAD.*    This  joint  task  group  and  their  subsequent 
activities  became  known  as  Project  SOAP. 

The  SOAP  group  received  support  and  encouragement 
from  NIAID's  Director,  Dr.  Dorland  J.  Davis,  who  arranged 
the  necessary  introductions  and  scheduled  meetings  with 
professional  staff  operations  personnel.   Thus,  the  group's 
entree  to  the  operational  levels  of  the  organization  was 
assured . 

1.3   Project  SOAP  Orientation 

Initially,  there  were  orientation  problems  to  resolve. 
Because  the  scope  of  the  assignment  and  the  objectives  of 
the  study  were  not  clearly  definable  in  advance,  NIAID's 
management  personnel  were  uncertain  about  what  to  expect  in 
terms  of  project  results.   A  portion  of  the  problem  lay 
with  the  Institute's  unfamiliarity  with  systems  analysis 
and  operations  research  terms  and  techniques.   Similarly, 
the  systems  analysis  people  from  TAD  were  unfamiliar  with 
biomedical  research.   Thus,  there  were  mutually  recognized 
difficulties  in  achieving  useful  results  within  the  allotted 
time.   Fortunately,  however,  there  was  also  a  sincere  resolve 
on  both  sides  to  make  a  meaningful  and  productive  start. 


*Robert  S.  Cutler,  Project  Leader,  and  Vincent  A.  Martino, 
Operations  Research  Analyst,  Technical  Analysis  Division, 
National  Bureau  of  Standards. 


In  the  beginning,  the  SOAP  task  group  spent  much  of  its 
time  interacting  with  various  key  members  of  the  NIAID 
organization.   The  discussions  were  quite  candid,  and  one 
charge,  frequently  reiterated,  emerged:   that  the  TAD  people 
were  lacking  in  biomedical  knowledge.   Acknowledging  this, 
the  TAD  analysts  pointed  out  in  response  that  the  real  issues 
of  concern  were  in  the  area  of  management  control  and  that 
expertise  and  "discipline"  in  this  field  was  what  they 
brought  to  the  project. 

1. 4   Purpose  and  Scope 

The  primary  aim  of  this  TAD/NIAID  study  was  to  explore 
the  application  of  systems  analysis  and  operations  research 
to  the  principal  management  problems  of  the  National 
Institute  of  Allergy  and  Infectious  Diseases,  particularly 
to  its  problem  of  allocation  of  resources. 

The  scope  of  the  assignment  included  two  operational 
divisions  within  the  Institute:   the  Collaborative  Program 
and  the  Extramural  Program.   (The  third  division,  the 
Intramural  Program,  was  not  included  in  the  Project  SOAP 
study  because  it  was  undergoing  a  reorganization  at  the 
time.)   Major  attention  was  to  be  given  to  the  Extramural 
Program,  which  is  responsible  for  awarding  research  project 
grants . 
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It  was  expected  that  this  assignment  would  also  create 
within  the  organization  the  broader  perspective  require&  to 
coordinate  the  management  of  some  of  the  Institute's  other 
biomedical  research  activities. 

Specifically,  the  joint  task  group  was  organized  to: 

1.  Review  the  process  for  awarding  Extramural 
Project  Grants  and  study  the  impact  of  the 
existing  allocation  procedure; 

2.  Develop  improved  selection  criteria  and 
management  methods  for  use  by  the  Institute's 
Staff,  its  management,  and  its  Advisory 
Council . 

It  was  anticipated  that  systems  analysis  techniques- - 
such  as  PERT  (Program  Evaluation/Review  Technique)*  for 
network  planning  of  Collaborative  research,  and  quantitative 
methods  for  measuring  research  accomplishments- -would  lead  to 
more  effective  management  procedures.   These,  in  turn,  would 
provide  a  basis  for  integrating,  within  NIAID,  closely  asso- 
ciated research  activities  into  Institute-wide  programs--an 
approach  already  under  serious  consideration  by  the  NIAID 
Director  at  that  time.   It  was  further  expected  that  if  the 
tools  provided  by  management  science  proved  effective  in 
handling  the  NIAID  Extramural  grants  program  they  might, 
similarly,  prove  effective  for  other  NIH  Institutes,  and 
perhaps  even  for  NIH  as  a  whole. 


^A  basic  tool  of  management  science  used  for  detailed  plan- 
ning and  control  of  complex  program  elements. 
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aThe  initial  work  assignment  for  the  Technical  Analysis 
Division  began  with  a  one  and  one-half  man-year  effort.   As 
the  project  became  operational,  additional  support  was  added. 
The  level  of  effort  in  the  first  year  was  $93,000.   The  pro- 
ject was  continued  in  June  1970  for  the  second  year  at 
$130,000,  to  which  an  additional  $33,000  was  added  during  the 
last  quarter  of  the  fiscal  period.   In  all,  approximately 
seven  man-years  and  a  total  of  $263,500  were  expended  during 
this  two-year  assignment. 

1.5   The  Initial  Task 

The  first  task  to  be  tackled  by  Project  SOAP,  before  any 
meaningful  systems  analysis  could  be  performed,  was  the 
development  of  a  primary  information  base.   They  had  to  find 
out  what  functions  the  various  operational  programs  were 
currently  performing;  what  kinds  of  information  key  decisions 
depended  upon;  and,  in  some  sort  of  logical  fashion,  to 
describe  what  this  "present  system"  consisted  of.   Each  part 
of  the  "system"  did  not  exist  in  isolation  but,  rather, 
within  a  context,  or  environment,  that  had  to  be  explicated. 
Elements  of  "environment"  were  originally  classified  as 
"internal"  (the  NIAID  context) ,  "external"  (the  NIH  context) , 
or  "Institutional"  (the  real  world  context) ,  a  term  which 
ultimately  describes  the  relationship  of  both  NIAID  and  NIH 
to  the  "outside  world." 
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In  setting  up  this  systems  framework,  organizing  the 
work  tasks,  and  isolating  the  kinds  of  problems  amenable  to 
solution,  the  task  group  found  that  flowcharts,  such  as  that 
illustrated  in  figure  1  (following  page) ,  were  useful  for 
providing  effective  communication.   They  provided  a  visual 
means  for  describing  the  process  under  investigation  and  for 
identifying  the  loci  of  specific  problems. 

The  management  problems  identified  in  the  NIAID  organi- 
zation could  be  isolated  thus:   1)  "operating  problems," 
i.e.,  problems  dealing  with  routine  internal  operating 
situations  (such  as  the  processing  of  approved  grants) ; 
2)  "strategic  problems,"  i.e.,  problems  relating  to  the 
anticipated  changes  in  the  external  environment  (such  as 
program  planning  and  evaluation);  and  3)  "policy  problems," 
i.e.,  problems  concerning  long-term  trends  in  the  NIH 
institutional  (outside  world)  context  (such  as  health 
research  priorities) . 

As  previously  stated,  the  primary  aim  of  Project  SOAP 
was  the  application  of  the  techniques  of  systems  analysis 
to  the  appropriate  "problems,"  once  the  problems  themselves 
were  identified.   Where  necessary  information  about  the 
operational  effectiveness  of  particular  programs  was  obscure 
or  even  lacking,  the  introduction  of  a  qualitative,  organi- 
zing framework  was  often  all  that  could  be  immediately 
provided;  but  even  such  first  steps  proved  far  from 
worthless . 
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The  "cost/effectiveness"  part  of  systems  analysis  was 
necessarily  deferred  in  favor  of  laying  a  foundation  for 
determining  more  precise  objectives  and  criteria,  and  for 
obtaining  better  bases  for  more  policy-relevant  data.   In 
the  case  of  the  Extramural  program,  this  kind  of  detailed 
examination  served  to  focus  the  Institute's  attention  on 
particular  information  gaps  in  its  primary  decision  process. 

1 . 6   Systems  Description 

The  primary  accomplishment  of  the  initial  three-month 
exploratory  phase  was  what  might  be  termed  an  "input- 
output-effect"  analysis  of  two  organization  divisions  of 
NIAID:   the  Collaborative  Program  and  the  Extramural  Program, 
The  results  obtained  were  functional  descriptions  of  the 
"existing  systems,"  which  formed  the  basis  for  identifying 
"problems"  and  formulating  requirements  for  the  "desired 
systems."   Also  prepared  were  descriptive  outlines  which 
clarified  the  meanings  of  words  used  by  the  people  making 
the  operational  decisions  in  each  program  and  which  docu- 
mented, in  operational  terms,  the  basic  activities  actually 
performed.   Included  were  carefully  worded  brief  statements 
of  program  goals,  objectives,  inputs/outputs,  and  associated 
performance  measures.   The  attempt  here  was  not  as  yet  to 
seek  to  make  improvements,  only  to  gather  the  kinds  of 
information  required  for  "Program  Management,"  in  order  to 
accurately  describe  the  decision  process. 
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For  example,  a  decision-oriented  information  flowchart 
(see  figure  2,  following  page)  was  prepared  to  describe  the 
existing  operations  of  the  Extramural  Program.   The  level  of 
detail  reflected  the  understanding  of  the  Institute's 
Associate  Director  for  that  program  of  what  kinds  of  opera- 
tional functions  were  both  necessary  and  sufficient  to 
define  his  principal  decision  processes  and  to  relate  the 
"inputs"  and  "outputs"  of  the  Extramural  Program  to  other 
organizational  components.   (A  detailed  description  of  these 
operations,  including  flowcharts,  is  contained  in  Project 
SOAP  Notes  No.  3  and  4.*) 

1 . 7   The  Desired  System 

As  the  current  operations  became  more  clearly  under- 
stood by  both  the  NIAID  managers  and  the  SOAP  task  group, 
the  next  question  to  be  answered  was:   What  ought  to  be  the 
"desired  system"?   Not  unexpectedly,  the  task  group  ran  into 
difficulties.   By  focusing  attention  on  the  problem  of 
"What  should  the  system  be  accomplishing"?   the  group  auto- 
matically raised  the  next  question:   "What  are  the  criteria 
for  measurement  of  performance?"   In  other  words,  they 
wanted  to  know  WHAT  the  organization  was  attempting  to 
achieve  and  HOW  it  actually  evaluated  its  degree  of  success. 


*"Present  System  Description,  Collaborative  Research  Pro- 
gram," Project  SOAP  Notes  No.  3,  December  15,  1969. 

"Extramural  Program:  Research  Grants  Award  System 
Description,"  Project  SOAP  Notes  No.  4,  January  30,  1970 
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And  these  questions  raised  additional  questions  about  WHOM 
the  Institute  serves  and  WHAT  are  the  consequences  of  its 
actions?- -all  important  concerns  to  a  management  responsible 
for  a  $100  million-a-year  enterprise. 

In  this  way,  the  SOAP  task  group  began  to  lay  the 
framework  for  an  Institute-wide  program  management  system 
(the  "desired  system") .   This  approach,  it  was  hoped, 
would  also  eliminate  many  of  the  existing  internal  communi- 
cations barriers  between  the  Intramural,  Extramural  and 
Collaborative  divisions  by  integrating  some  of  their 
separate  efforts  toward  a  common  goal:   the  development  of 
biomedical  research  programs  designed  to  be  more  responsive 
to  national  health  research  needs. 

The  attainment  of  this  paramount  goal,  it  was  realized, 
would  require  considerable  effort.   The  Institute's  basic 
management  functions- -planning ,  organizing,  and  evaluating-- 
would  have  to  be  strengthened,  and  formal  methods  and 
techniques  to  assist  decision-making  at  various  levels  would 
have  to  be  developed. 

More  specifically,  the  fundamental  management  problem 
confronting  NIAID  was  "allocation  of  resources."   The 
Institute  was  searching  for  a  quantitative  method  for 
assessing  the  relevance  of  both  its  major  research  programs 
and  its  individual  project  proposals  in  terms  of  its 
explicit  public  health  mission.   During  the  initial  task 
group  orientation,  for  example,  the  following  memorandum, 

18 


by  the  Assistant  Scientific  Director  for  Collaborative 
Research  to  his  staff,  described  the  type  of  management 
system  desired  by  the  Institute  for  evaluating  its 
Collaborative  Research  Programs  in  terms  of  NIAID 
objectives : 

a)  Collaborative  programs  should  be  extensions 
of  Intramural  Research  at  NIH,  to  assure  the 
availability  of  competent  Project  Officers. 

b)  Collaborative  programs  should  be  coordinated 
with  the  Extramural  and  Intramural  Programs 
of  NIAID,  with  other  NIH  Institutes  and 
Government  agencies,  and  with  any  similar 
research  activity  in  the  private  sector. 
Where  major  technological  advances  are 
sought,  the  interests  of  prospective  user 
organizations  should  be  an  important 
consideration. 

c)  Full  use  of  Government,  industry,  and  academic 
scientists  should  be  made  in  obtaining  advice 
and  assistance  in  defining  the  scientific 
base,  goals,  and  feasibility  of  new  research 
programs . 

d)  Programs  must  have  major  Institute  review  and 
evaluation  at  intervals  not  to  exceed  five 
years . 

e)  Contract  Review  Committees  are  charged  with 
assessment  of  technical  proposals,  contractor's 
competence  and  facilities,  and  reasonableness 
of  budgetary  cost  and  time  estimates.* 

Furthermore,  because  of  the  large  number  of  research 

proposals  which  come  to  NIAID  in  the  form  of  Extramural 

Grant  applications,  or  Collaborative  contract  proposals,  or 

through  the  research  interests  of  senior  scientists  on  its 

Intramural  Staff,  NIAID  desired  a  more  systematic  means  for 


*Memorandum,  Dr.  John  R.  Seal  to  Dr.  Dorland  J.  Davis, 
Director  NIAID,  "Role  of  Scientific  Director,  Advisory 
Committees,  Branch  Chiefs  in  Collaborative  Research," 
February  9,  1970. 
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judging  what  it  termed  "program  relevance":   the  importance 
to  the  Institute  of  supporting  a  particular  project  proposal. 
It  believed  that  a  quantitative  method  for  evaluating  pro- 
gram performance  in  general  would  offer  the  potential  for 
improving  its  current  process  for  selecting  research  pro- 
jects and  establishing  priorities.   The  sheer  volume  and 
complexity  of  this  type  of  decision-making  defied  intuitive 
attempts  to  arrive  at  an  optimum  solution.   Nevertheless, 
definite  choices  and  hard  trade-offs  were  being  made-- 
albeit,  under  tacit  assumptions  about  what  was  and  was 
not  "relevant." 

Therefore,  the  Project  SOAP  task  group  set  forth  to 
create  an  awareness  of  what  a  systems  analysis  capability 
within  the  NIAID  Institute  could  do  for  facilitating  the 
development  of  improved  management  techniques  while,  at 
the  same  time,  promoting  the  wider  acceptance  of  such  new 
procedures  throughout  the  Institute.   While  there  were  some 
signs  after  the  first  year  (June  1971)  that  significant 
changes  in  NIAID' s  management  philosophy  had  taken  place, 
the  ultimate  effectiveness  of  the  recommended  DAPA  procedure 
developed  by  the  SOAP  task  group  has  yet  to  be  realized. 
The  Institute  has  recently  (March  1972)  begun  to  emphasize  the 

planning  and  evaluation  of  overall  programs  (rather  than  the 
mere  review  of  the  details  of  individual  projects),  and  it 


ft 


*DAPA:  Dollar  Allocation  by  Program  Area- -to  be  fully 
explained  in  the  following  pages. 
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has  apparently  gained  confidence  in  entering  more  actively 
into  the  proposed  Dollar  Allocation  method  for  guiding 
investment  in  socially  relevant  areas  of  health  research. 

To  come  as  far  as  it  did,  Project  SOAP  used  an  approach 
which  involved  three  major  phases--an  Exploratory  Phase, 
a  Diagnostic  Phase,  and  a  Prescriptive  Phase--which  are 
described  in  the  following  sections  of  this  report. 
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2.   THE  EXPLORATORY  PHASE 
The  Project  SOAP  task  group  commenced  its  assignment  with 
a  review  of  the  Collaborative  Program.   This  operational  pro- 
gram consists  chiefly  of  target-oriented  research  projects 
funded  by  contracts;  it  also  involves  some  selected  Intra- 
mural laboratory  activities.   It  was  believed  that  the 
techniques  of  systems  analysis- -for  instance,  Program 
Evaluation  and  Review  Technique  (PERT)  and  Critical  Path 
Method  (CPM)  Network  Planning  techniques  for  organizing 
time-dependent  and  information-dependent  activities,  and  the 
development  of  briefing  charts  for  progress  display  purposes-- 
would  result  in  a  more  effective  program  management  procedure. 

The  second  undertaking  involved  the  Extramural  Program. 
It  included  the  development  of  an  operational  flowchart  (see 
previously  mentioned  figure  2)  and  studies  of  the  implica- 
tions of  the  "Decile  System"*  and  of  the  current  allocation 
procedure  for  awarding  research  project  grants,  including  an 
investigation  of  criteria  for  selecting  Special  Emphasis  pro- 
jects.  The  objective  was  more  explicit  criteria,  leading  in 
turn  to  improved  operating  procedures . 

A  brief  management  overview  of  the  NIAID  Institute  and 
its  operating  activities  and  procedures  is  given  next,  as 
background  for  the  subsequent  accounts  of  the  two  specific 
tasks  sketched  above. 


*A  ranking  method  for  equalizing  the  distribution  of  project 
grant  priorities  recommended  by  the  various  Study  Sections. 
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2. 1   The  Institute's  Organization 

The  operations  of  NIAID  are  divided  into  three  major 
divisions:   the  Intramural  Program,  which  includes  the 
research  conducted  by  the  in-house  laboratories;  the 
Extramural  Program,  which  is  concerned  primarily  with 
research  and  training  grants  awarded  to  individual 
outside  investigators;  and  the  Collaborative  Program, 
which  directs,  under  contract,  research  with  industry 
and  other  institutions.   To  put  it  another  way,  the 
operations  are  divided  into  three  major  organizational 
activities:   General  Laboratories  and  Clinics 
(Intramural) ,  Research  and  Training  Grants  (Extramural) , 
and  Nationally  Organized  Research  Programs  (Collaborative) 
Below  are  more  fully  expanded  functional  descriptions  of 
the  Collaborative  and  Extramural  programs  as  they 
existed  when  the  SOAP  Task  Group  was  first  introduced 
to  them  in  October  1969. 

2.1.1   Collaborative  Research  Program  Operations 

The  Collaborative  Research  (and  development) 

Program  is  one  of  the  three  major  operational  units  of 

the  Institute.   It  differs  from  the  others  in  that  its 

operations  are  conducted  primarily  through  contracts 

with  outside  companies  and  research  laboratories.   Its 

role  is  to  direct,  coordinate,  and  provide  support  for 

developmental  research  on  specific  biomedical  products 

and  related  scientific  information. 
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The  Collaborative  program's  major  purpose  is  the 
translation  of  the  findings  of  basic  research  (biomedical 
knowledge)  into  methods  or  products  for  the  treatment  and 
control  of  allergic  and  infectious  diseases.   Specific 
objectives  are:   the  development  and  testing  of  specific 
vaccines  against  viral  and  bacterial  agents;  the  production, 
control  and  distribution  of  biomedical  research 
reference  reagents;  and  the  development  of  methods  and 
knowledge  about  various  types  of  human  tissues  and  their 
compatibility  with  other  tissues  for  use  in  transplantation 
of  human  organs.   In  addition,  the  members  of  the  staff 
provide  support  and  scientific  management  for  two  inter- 
national programs:   a  cooperative  medical  program  with 
Japan,  and  a  domestic  research  and  training  program 
with  an  international  component. 

These  collaborative  efforts  are  conducted  via 
the  mechanism  of  research  contracts  with  industry, 
universities,  federal  agencies,  and  other  biomedical 
organizations.   They  are  directed  by  members  of  the 
NIAID  scientific  and  administrative  staff,  with  advice 
and  counsel  obtained  from  various  ad  hoc  advisory 
groups  . 
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The  Contract  Review  Group  (CRG)*,  composed  of  the 
top  management  team,  review  each  contract  proposed  for 
the  development  of  specific  outputs,  such  as  new 
experimental  vaccines.   They  examine  the  basic  data 
and  contractors'  proposals  for  development  and  production, 
and  then  approve  those  proposals  which  they  find  best  meet 
the  Institute's  needs  in  that  area,  such  as  the  development 
of  a  new  vaccine. 

2.1.2   Extramural  Research  Program  Operations 

The  Extramural  Research  Program  provides  support, 
through  the  mechanism  of  research  grants,  to  public 
and  other  non-profit  institutions  and  investigators 
to  establish,  expand,  and  improve  research  activities 
in  the  health  sciences  and  related  fields. 

The  existing  process  for  the  award  of  Extramural 
research  grants  begins  when  applications  are  received 
by  NIH  from  prospective  research  investigators.   Each 
application  is  assigned  by  the  NIH  Division  of 
Research  Grants  (DRG)  to  a  categorical  Institute 


*Contract  Review  Group.   Its  members  are  the  Institute 
Director,  the  Scientific  Director,  the  Assistant 
Scientific  Director,  the  Associate  Director  for 
Collaborative  Research,  the  Associate  Director  for 
Extramural  Programs,  the  Special  Assistant  to  the 
Director,  the  Executive  Officer,  and  the  Chief  of 
Program  Planning. 
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(such  as  NIAID)  whose  mission  most  closely  involves  the  pro- 
posed research  activities,  and  to  one  of  55  Study  Sections 
(discipline-oriented  Peer  Review  Groups)  for  review  and 
assessment.   This  process  for  selecting  recipients  for  pro- 
ject grants  from  NIH  is  widely  known  and  accepted  by  many 
segments  of  the  scientific  community.   It  is  similar  in 
principle  to  that  used  by  other  Government  agencies 
supporting  basic  research,  like  the  Atomic  Energy  Commission 
and  the  National  Science  Foundation. 

(As  previously  stated,  the  Project  SOAP  task  group 
did  not  examine  the  activities  of  the  Intramural  Program 
because  it  was  undergoing  a  reorganization  at  the  time.) 

The  number  of  programs  within  these  three  organiza- 
tional divisions- -  the  Intramural,  the  Collaborative,  and  the 
Extramural- -could  total  anywhere  from  ten  to  about  fifty, 
depending  on  the  enumerator's  choice  among  various  meanings 
of  the  word  "program."   Activities  wholly  within  the 
Collaborative  Research  Program,  such  as  the  Vaccine 
Development  Program,  or  the  Pneumococcal  Vaccine  Program, 
or  the  Research  Reference  Reagents  Program,  etc.,  were 
frequently  spoken  of  simply  as  "programs."   In  the 
Extramural  organization  there  were  research  activities 
supported  by  grants  in  loosely  defined  areas  called  Special 
Emphasis  Research  Programs,  such  as  Transplantation  Immuno- 
logy, Infectious  Hepatitis  and  Antiviral  Substances.   The 
fact  was  that  a  kind  of  vagueness  about  many  of  the  terms 
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used  by  members  of  the  Institute- -terms  such  as  "program"- - 
accounted  for  the  serious  semantic  difficulties  the  SOAP 
group  encountered  while  attempting  to  formalize  precise 
"systems  descriptions." 

2 . 2   Grant  Allocation  Procedure 

The  allocation  of  resources  to  Extramural  project 

grants  had  been  a  major  concern  of  the  NIAID  Institute  for 

a  number  of  years.   Since  1962,  the  "decile  system"  has 

been  used  by  the  Institute  as  the  basis  for  allocating 

grants.   It  employs  a  decile-ranking  method  for  equalizing 

the  distribution  of  priorities  recommended  by  some  ten 

to  twenty  different  Study  Sections.   Following  is  a  more 

detailed  description  of  the  system  as  employed  by  NIAID. 

THE  DECILE  SYSTEM  OF  THE  NIAID 

Since  1962,  the  National  Institute  of  Allergy  and 
Infectious  Diseases  has  used  a  "decile"  system  to 
equalize  the  differing  levels  of  priorities  on 
research  grant  applications  recommended  by  different 
Study  Sections.   The  details  of  this  system  and 
some  of  its  effects  are  described  below. 

In  accordance  with  the  policy  of  the  National 
Institutes  of  Health,  every  Study  Section  provides 
a  priority  score  for  each  application  recommended 
for  approval  by  the  majority  of  its  participating 
members  (for  grants  assigned  to  NIAID  for  payment) . 
This  priority  score  is  derived  in  the  following 
manner : 

1.   Each  participating  Study  Section  member 
rates  every  approved  application  on  the 
basis  of  order  of  payment  from  1  (first 
order)  to  5  (last  order) .   Criteria  used 
in  making  this  judgment  include  scientific 
merit,  study  design,  originality,  competence 
of  investigator,  and  adequacy  of  working 
environment . 
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2.  After  the  meeting,  scores  for  each  appli- 
cation are  averaged  and  the  result  multiplied 
by  100  to  provide  a  3-digit  rating. 

3.  The  resulting  priority  score  is  recorded  on 
a  confidential  "resume"  of  actions  of  the 
Study  Section  and  is  distributed  to  the 
awarding  Institutes/Divisions. 

When  such  a  "resume"  is  received  by  the  NIAID,  all 
priority  scores  for  that  Study  Section  are  listed  in 
numerical  order,  regardless  of  the  Institute  to 
which  assigned.   The  list  is  then  divided  into 
10  equal  parts  known  as  "deciles,"  with  the  first 
decile  representing  the  most  favorable  priority. 
If  an  application  ties  with  another  or  falls 
between  deciles,  it  receives  the  more  favorable 
rating.   The  decile  number  is  then  marked  on  each 
summary  sheet  sent  to  the  Council.   Applications 
falling  in  the  10th  decile  are  not  paid  unless  the 
priority  is  raised  by  specific  vote  of  Council. 

Throughout  a  fiscal  year,  awards  are  made  on  grants 
which  fall  in  the  most  favorable  deciles  to  the 
financial  level  that  can  be  maintained  throughout  the 
year.   Renewal  grants  in  the  least  favorable  deciles 
are  brought  to  a  termination  and  those  in  the  inter- 
mediate deciles  are  extended  with  the  expectation  that 
they  will  be  renewed  later  in  the  year  if  sufficient 
funds  become  available.   After  the  March  NAAIDC 
meeting,  applications  in  less  favorable  deciles  from 
this  and  prior  Council  meetings  are  awarded  in  order 
as  long  as  funds  are  available.   Unless  an  applicant 
withdraws  a  pending  application,  it  remains  in 
competition  for  available  funds  until  the  end  of  the 
fiscal  year.* 

Various  methods  for  managing  the  allocation  of  grants 
had  been  proposed,  and  the  one  in  use  partially  reordered 
the  Study  Section- imposed  priority  ranks  of  approved  appli- 
cations in  accordance  with  Institute  preference  criteria, 
termed  "program  relevance."   This  method  had  not  proven 
entirely  satisfactory  to  all  members  of  the  Advisory  Council. 


*See  also  footnote  (*) ,  p.  34. 
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Selected  grant  proposals  in  Special  Emphasis  Research 
categories  were  given  an  MHPR"*  rating,  resulting  in  an 
arbitrary  two-decile  escalation  and  often  raising  them  to 
payment  status.   This  practice  (facetiously  described  as 
"decile-diddling")  effectively  introduced  the  Institute 
Staff's  program  preferences  but  did  not  provide  the  Advisory 
Council  with  the  kinds  of  information  that  several  members 
felt  were  needed  "to  make  responsible  recommendations."** 

2 . 3   Program  Relevance  Concept 

It  was  primarily  this  dilemma- -whether  to  use  Study 
Section  priority  scores  "as  is"  or  to  modify  them  by  the 
Institute's  program  relevance  ratings  in  order  to  determine 
payment  priorities- -that  led  the  Institute  Director  to 
appoint  a  special  subcommittee  of  the  Advisory  Council  to 
look  into  the  matter.   Specifically,  the  Criteria  Sub- 
committee was  to  develop  an  acceptable  concept  of  "relevance" 


*HPR:   A  rating  given  a  particular  project  considered  by  the 
Institute  to  be  of  "High  Program  Relevance."   The  use  of 
preferential  payment  designations  "HPR,"  "LPR,"  and  "NPR" 
evolved  over  several  Council  meetings  between  the  years 
1965  and  1969.   As  the  effects  of  such  reordering  of 
Study  Sections'  priority  recommendations  (two  decile 
levels,  up  or  down)  were  felt  by  the  scientific  community, 
and  reacted  to  by  the  Advisory  Council,  steps  toward  modi- 
fication of  this  procedure  were  taken.   Finally,  at  the 
March  1970  Council  meeting,  a  motion  was  carried  to  dis- 
continue the  LPR  and  NPR  ratings  until  the  basic  questions 
concerning  the  staff's  preferential  payment  criteria  could 
be  clarified  and  an  improved  operating  procedure  adopted. 

**Dr.  Churchman,  November  1970  Council  Meeting. 
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and  establish  explicit  criteria  for  use  in  judging  factors 
associated  with  "relevance."   In  addition,  they  were  to 
consider  an  improved  relevance  rating  procedure. 

The  plain  fact  of  the  matter  was  that  the  Study  Section 
members  themselves  realized  that  all  their  time  and  effort 
spent  on  reviewing  a  particular  grant  in  a  research  area 
might  come  to  naught;  regardless  of  its  scientific  merit,  a 
particular  aspect  of  research  might  not  get  funded  if  it 
were  not  deemed  "relevant"  to  some  mission.   Similarly,  they 
realized,  nationally  recognized  scientists  might  not  receive 
support  for  work  they  considered  extremely  important  to  a 
particular  area  because  their  work  was  judged  "non-relevant," 
while  inferior  scientists  might  receive  support  because  their 
work  was  judged  "more  relevant." 
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3.   THE  DIAGNOSTIC  PHASE 
The  purpose  of  this  phase  was  the  diagnosis  of  key 
issues  requiring  policy-level  resolution  before  an  analysis 
of  possible  operational  changes  could  be  made. 

The  following  is  a  brief  outline  of  Project  SOAP's 
identification  of  three  principal  factors  associated  with 
the  management  of  the  Extramural  Program: 
Study  Section  Priority  Scores 
Program  Relevance  Criteria 
Preferential  Payment  Procedure 

3. 1   The  Preferential  Payment  Issue 

In  1966,  when  the  NIAID  originally  presented  its  plan 
for  stimulating  and  supporting,  preferentially  through 
grants,  more  original  investigations  bearing  on  disease 
problems  deemed  of  special  importance,  its  advisory 
committees  (particularly  its  Advisory  Council)  gave  consid- 
eration to  ways  in  which  such  a  programmatic  structure  could 
be  achieved  within  the  Extramural  Program. 

The  following  year,  the  Institute's  staff,  with  the 
help  of  its  Advisory  Council,  identified  several  areas  of 
"high  program  relevance"  which  they  began  to  call  Special 
Emphasis  Research  Programs.   They  felt  these  fields  of 
biomedical  research  were  deserving  of  special  emphasis 
within  the  context  of  the  Extramural  Program.   At  that  time, 
these  areas  comprised  about  35  percent  of  grant  projects  and 
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funds.   They  were  judged  to  be  of  greater  importance  to  the 
health  needs  of  the  nation  than  other  areas.   The  intention 
clearly  was  not  to  limit  or  to  direct  original  scientific 
research  supported  by  NIAID  grants  but,  rather,  to  give 
scope  to  individual  investigators  in  pursuing  their  preferred 
lines  of  research  in  a  way  which  would  foster  advancement  on 
disease  problems  of  greatest  national  importance.* 

The  challenge  was  clear,  but  the  mechanism  was  not,, 
especially  during  the  subsequent  fiscal  retrenchment  which, 
for  the  next  several  years,  resulted  in  making  payment 
priority  decisions  difficult  for  the  Institute. 

In  the  opinion  of  several  outspoken  members  of  the 
Advisory  Council,  the  review  of  any  publicly-funded  program 
requires,  at  the  outset,  clarification  and  agreement  on  a 
statement  of  program  goals  and  priorities.   These  individuals 
have  pointed  out  that  such  a  requirement  is  of  particular 
importance  when  budgets  are  limited  and  apparently  worthy 
efforts  must  go  unfunded. 

A  substantial  number  of  Study  Section  participants  in 
the  NIAID  Extramural  grants  award  process  felt,  however,  that 
considerations  other  than  intrinsic  scientific  quality  ought 
not  to  influence  research  grant  payment.   This  was  confirmed 


*Dorland  J.  Davis,  "Remarks  on  Special  Emphasis  Programs  of 
the  National  Institute  of  Allergy  and  Infectious  Diseases," 
The  Journal  of  Infectious  Diseases,  University  of  Chicago, 
Vol.  121,  No.  2,  February  1970,  p.  231. 
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by  the  study  of  rating  criteria  in  Exercise  II.*   One 
of  the  basic  difficulties  appeared  to  lie  in  an  uncertainty 
about  desired  "ends,"  a  dilemma  which  is  associated  with  the 
current  debate  between  applied  research  vs.  basic 
research.   Questions  raised  by  this  situation  can  perhaps 
be  phrased  thusly:   Is  the  Advisory  Council  expected  to 
develop  planning  criteria,  provide  judgment  on  program 
balance,  and  evaluate  program  implementation,  or  is  it 
merely  to  reassure  the  Institute's  Director  that  the 
Extramural  program  is  moving  along  the  right  track? 

3.1.1   Study  Section  Priority  Scores 

It  became  a  matter  of  concern  to  members  of  NIAID's 
National  Advisory  Council  that  different  interpretations 
were  being  given  by  different  Study  Sections  to  their 
"scientific  merit"  judgments  and  resulting  priority 


*Exercise  II,  "Biomedical  Research  Relevance  Criteria," 
Project  SOAP,  National  Bureau  of  Standards  Report  No. 
10-423,  June  30,  1971.   This  Exercise  will  be  explained 
in  the  following  pages . 
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scores.*   Study  Sections  were  never  required  to  provide 
explicit  advice  on  order  of  payment,  but  practical  con- 
siderations in  some  NIH  Institutes  made  such  rating  scores 
useful  for  payment  priority  purposes. 

The  priority  scores  were  ostensibly  based  on  con- 
siderations of  "scientific  merit,"  but,  in  reality,  the 
scores  reflected  the  Study  Section  members'  personal 
notions  of  the  extent  or  degree  to  which  the  proposal 
should  be  supported  by  extramural  funds.   (An  earlier 
study  of  NIH  grant  application  procedures  by  Saunders  and 
Gordon**  concluded  that  Study  Section  priority  scores  had 
become  "a  haphazard  mixture  of  two  separable  considerations" 


*A  priority  score  was  traditionally  derived  in  the  following 
manner: 

1.  Each  participating  Study  Section  member  rated 
every  "approved"  grant  application  on  the  basis 
of  "scientific  merit"  from  1.0  (highest)  to 
4.5  (lowest)  in  0.5  increments. 

2.  These  two  "digit  ratings  for  each  application 
were  averaged  and  the  result  multiplied  by  100 
to  provide  a  three-digit  numerical  score. 

3.  The  resulting  "priority  score"  was  recorded 
on  a  confidential  resume  of  action  by  the 
Study  Section  and  distributed  to  the  previously 
designated  NIH  Institute  for  funding. 

An  example:   Each  of  six  Study  Sections  assigns 
"scientific  merit"  scores  1.5,  2.5,  2.0,  1.5,  2.0,  2.5. 

Priority  Score  =  \   (1.5  +  2.5  +  2.0  +  1.5  +  2.0  +  2.5) 

x  100  =  12-°  x  100  =  200. 
6 

**Saunders,  J.  Palmer  and  Gordon,  Mordecai,  H. ,  "NIH  Study 
Section  Ratings:  Scientific  Merit  or  Order  of  Payment," 
National  Cancer  Institute  (1965) . 
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referring  respectively  to  the  evaluation  of  research  grant 
proposals  for  scientific  "quality"  and  for  "order  of  payment." 
Their  hypothesis  of  separability  was  inferred  statistically 
from  the  scores  of  four  Study  Sections  who  reviewed  225 
individual  Extramural  research  grant  applications  assigned 
to  the  National  Cancer  Institute  in  1965.) 

The  fact  of  the  matter  was  that  the  components  of 
the  concept  of  "scientific  merit"  as  applied  by  the  Study 
Sections  were  not  explicitly  known.   The  resulting  confusion 
raised  the  question  of  the  appropriateness  of  certain  other 
criteria  in  this  decision  process  and  thus  the  desirability 
of  using  multiple  criteria  judgments  as  the  basis  for 
determining  grant  payment  priority. 

3.1.2   "Program  Relevance"  Criteria 

During  the  March  1970  Advisory  Council  Meeting, 
several  members  voiced  concern  about  how  decisions  were 
made  on  matters  pertaining  to  "program  relevance."   They 
wanted  to  know,  specifically,  what  "criteria"  were  used 
for  determining  HIGH,  LOW,  or  NO  "program  relevance." 
A  motion  was  made  and  carried  by  the  majority  that  the 
"Council  not  consider  Low  Program  Relevance  (LPR)  or  NO 
Program  Relevance  (NPR)  ratings  in  evaluating  payment 
of  grants."*   This  meant  that  the  Council  would  no 
longer  consider  these  special  cases  until  further  informa- 


*Minutes  of  March  1970  meeting,  NIAID  Council,  March  18,  1970 
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tion  was  provided  about  the  criteria  used.   "HPR"  was 
allowed,  however,  in  order  to  provide  in  the  interim  some 
expression  of  Institute  payment  preference. 

3.1.3   Preferential  Payment  Procedure 

Normally,  the  NIH  Extramural  grant  review  procedure 
is  explicitly  concerned  with  the  "scientific  merit"  of  the 
proposals,  with  other  aspects  represented  only  implicitly. 
However,  biomedical  research  supported  by  NIAID  and  the 
other  Institutes  of  NIH  affects  ultimately  the  health  and 
well-being  of  the  entire  American  people.   Realization  of 
this  fact  has  underscored  legislation  concerning  NIH 
for  over  25  years,  with  confirmation  of  benefits  from  several 
viewpoints:   physical,  social  and  economic*   Both  fundamental 
and  applied  research  produce  new  knowledge  which  can  make 
immediate  or  long-range  contributions  toward  the  effective 
control  of  certain  diseases  and  associated  health  problems. 

Grant  proposals  selected  for  the  Special  Emphasis 
Research  categories  were  given  an  "HPR"  rating,  resulting 
in  an  arbitrary  two-decile  escalation,  which  often  raised 
them  to  payment  status.  This  practice  introduced  the 
Institute's  preferences  ("program  relevance")  but  did  not 
provide  the  Advisory  Council  with  the  kind  of  information 
it  felt  was  needed  before  it  could  make  responsible 


*See  Stephen  P.  Strickland,  "Policy  Making  in  Biomedical 
Research:   A  Perspective  on  Relevance,"   Project  SOAP 
Notes  No.  16,  March  15,  1971. 
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recommendations.   For  example,  the  Council  wanted  to  know 
the  effect  of  such  re-ranking  action  on  the  other  competing 
grants,  the  relative  "importance"  to  the  Nation's  health 
of  each  Special  Emphasis  area,  and  the  rationale  or  plan 
behind  each  selection  of  such  an  area.   The  task  group 
set  out  to  satisfy  these  needs. 

3. 2   Preliminary  Study  of  "Health  Relevance"  Assessment 
(Exercise  I) 

In  June  1970,  Project  SOAP  conducted  a  limited 
investigation  among  the  three  members  of  the  Advisory 
Council  Criteria  Subcommittee:   Drs .  C.  West  Churchman, 
Irwin  C.  Gunsalus,  and  Maxwell  I.  Wintrobe.   They 
agreed  to  try  to  assess  "health  relevance"  as  a  concept 
distinct  from  "scientific  merit."   Twenty  applications 
were  sent  by  mail  to  each  of  the  three  Criteria 
Subcommittee  members .   Their  review  was  made  without 
knowledge  of  the  "scientific  merit"  priority  scores,  and 
without  any  discussion  or  communication  between  the 
three  members . 

The  results  of  this  study,  based  on  a  cursory  analysis 
of  the  twenty  "relevance"  rating  scores,  indicated  that  it 
was  possible  for  Advisory  Council  members  to  attach  a 
"health  relevance"  category  to  proposals,  and  that  there 
appeared  to  be  some  interesting  relationships  between  the 
scores  of  this  Ad  Hoc  Group  and  the  NIH  Study  Sections' 
priority  scores.   For  example,  comparison  of  these 
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relevance  scores  with  the  priority  scores  for  the  same 
proposals  showed  that  eight  out  of  11  judged  "high"  for 
"scientific  merit"  scored  "high"  for  "health  relevance," 
and  four  out  of  six  judged  "low"  for  "scientific  merit" 
scored  "low"  for  "health  relevance."   The  consensus  of 
the  group  was  that,  in  spite  of  the  lack  of  specified 
criteria,  there  appeared  to  be  some  commonly  held  basis 
for  assessing  "relevance." 

These  findings  were  presented  by  the  chairman  of 
the  Criteria  Subcommittee  to  the  NIAID  Advisory  Council 
at  its  June  1970  meeting.   This  led  to  the  Council's 
recommending  an  expansion  of  the  study  of  relevance 
criteria  (Exercise  II),  with  instructions  to  include 
a  more  representative  sample  of  research  grant  applications 
and  a  wider  participation  by  other  qualified  reviewers . 

3 . 3   Biomedical  Research  Relevance  Criteria  (Exercise  II) 

This  more  comprehensive  exploratory  investigation  was 
made  in  September  1970  in  an  attempt  to  improve  the  general 
understanding  of  the  decision  process  for  awarding  Extramural 
research  project  grants  at  NIAID.   Exercise  II  involved  a 
dual  review,  the  rating  of  research  proposals  separately, 
first,  for  "scientific  merit"  and,  second,  for  "health 
relevance;"  and  it  attempted  to  identify  the  various 
decision  criteria  used  by  different  groups  within  the 
NIH  Extramural  grants  award  system. 

38 


A  total  of  120  professionals,  most  of  whom  were 
directly  involved  in  the  system  of  awarding  grants,  were 
grouped  into  six  "juries"  for  this  exercise.   Each  partici- 
pant rated  a  sample  of  24  research  applications  (or  sub- 
sample  of  that  24)  on  dimensions  of  "scientific  merit"  and/or 
"health  relevance."   These  ratings  were  analyzed  to  determine 
if  there  existed  a  meaningful  and  consistent  concept  under- 
lying these  dimensions  and,  if  so,  what  the  characteristics 
of  this  concept  were.   Corollary  evidence  was  obtained  from 
statements  of  criteria  elicited  from  some  of  the  participants. 

Analyses  of  jury  mean  scores  (using  nonparametric 
correlation  techniques)  indicated  that  there  were  statisti- 
cally significant  similarities  among  juries  in  scores  associ- 
ated with  "health  relevance"  for  the  sample  applications 
rated.   One  jury,  composed  of  six  Study  Sections,  both 
reported  and  exhibited  its  inability  to  distinguish  clearly 
between  assessments  for  "health  relevance"  and  those  for 
"scientific  merit."   The  general  consistency  among  the 
"health  relevance"  ratings  and  their  agreement  with  the 
"scientific  merit"  ranking  suggested  that  the  Study  Sections' 
judgments  about  priorities  were  generally  related  to  their 
concept  of  "scientific  merit,"  at  least  on  these  test  appli- 
cations.  Characteristics  of  the  criteria  stated  by  individ- 
ual jurors  as  their  bases  for  judgment,  moreover,  documented 
the  existence  of  review  considerations  other  than  scientific 
quality. 
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The  results  of  this  exercise  were  not  definitive. 
However,  certain  characteristics  were  indicated.   There 
was  general  although  not  unanimous  agreement  among  reviewers 
about  their  concepts  of  "health  relevance."   (The  lack  of 
unanimity  was  not  surprising,  since  the  concept  of  "health 
relevance"  is  a  complex  one  requiring  greater  specificity 
to  assure  a  common  understanding.)   The  concept  underlying 
"scientific  merit"  as  interpreted  by  Study  Sections  included 
more  than  simple  scientific  quality  considerations.   This 
additional  dimension  was  closely  related  to  their  concept 
of  "order  of  payment." 
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4.   THE  PRESCRIPTIVE  PHASE 
The  accomplishment  of  this  phase  was  the  formulation  of 
specific  guidelines  and  recommendations  leading  to  an 
improved  Program  Management  discipline  for  NIAID.   Several 
mutually  related  actions  pertaining  to  resource  allocation 
and  NIAID  Program  Management  were  recommended. 

4. 1   The  Program  Management  Concept 

The  system  recommended  for  managing  the  various  pro- 
grams, projects,  and  activities  within  the  Institute  was 
shown  in  comparison  to  the  then-present  mode  of  operation 
(see  figure  3,  following  page).   The  principal  attributes  of 
the  existing  system  were  characterized  as  being  detailed, 
retrospective,  and  custodial  in  nature.   By  comparison,  the 
proposed  management  system  provided  a  broader  perspective  and 
depended  upon  anticipatory  planning  and  the  use  of  explicit 
measures  of  performance.   This  was  considered  to  be  a  desir- 
able improvement  by  many  Council  members  and  Institute  Staff. 

Under  the  existing  system,  the  cost/effectiveness 
relationships  were  highly  subjective.   Moreover,  the  payment 
priority  advice  was  received  from  many  individual  Study 
Sections  that  might  not  have  had  appreciation  of  the 
Institute's  major  health  research  programs.   The  basic 
improvement  in  resource  allocation  desired  by  the  Institute 
required  a  means  for  maintaining  some  proper  balance  between 
its  two  primary  mission  goals:   the  support  of  basic  science, 
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and  the  achievement  of  targeted  research.   In  addition, 
there  was  need  for  a  consistent  method  for  communicating 
the  Institute's  programs,  plans,  and  performance  to  others 
(inside  and  outside) ,  while  also  providing  a  more  rational 
basis  for  decision-making,  particularly  for  the  Extramural 
Grants  Program. 

Furthermore,  to  remedy  the  Institute's  problem  in 
determining  a  preferred  order  of  payment  for  grants, 
Project  SOAP  recommended  a  management  concept  for 
coordinating  the  planning,  budgeting,  and  resource 
allocation  decisions  in  a  way  that  would  provide  the 
Institute  greater  control  over  its  program  performance. 

The  resource  allocation  problem  was  reduced  to  three 
sets  of  decisions  within  the  Institute:   1)  the  program 
planning  decision,  2)  the  "scientific  merit"  rank 
decision,  and  3)  the  program  category  assignment  decision. 
The  effect  was  to  delegate  to  the  operational  level  the 
authority  to  judge  the  relevance  or  contribution  of  a 
particular  research  project  proposal  and  to  assign  it 
to  a  proper  program  category.   These  categories  were 
the  ones  associated  with  a  prior  Council  commitment  of 
funds  to  a  Special  Emphasis  Research  area  in  which  the 
ultimate  solution  of  a  particular  disease  problem  was 
being  sought. 
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Progress  toward  such  health-oriented  goals  could  be 
assessed  by  the  estimated  reduction  in  time  and  resources 
remaining  to  reach  a  level  of  scientific  knowledge  which, 
in  the  opinion  of  experts,  is  sufficient  to  transfer  the 
project  to  the  Intramural/Collaborative  Program  (supported 
by  other  funds).   In  the  event  such  progress  lagged,  a 
decision  could  be  made  to  abandon  the  particular  approach 
by  phasing  out  investment  in  it.   In  either  case,  the 
appropriate  funding  (transfer  or  abandonment)  would  call 
for  Council/Staff/Institute  interaction. 

While  this  summation  is  a  simplification  of  the 
Institute's  program  management  role,  it  clearly  indicates 
a  resolution  of  the  major  portion  of  the  dissonance  in 
the  current  "dual-review"  system  which  stemmed  from  the 
failure  to  define  and  delegate  clear  functions  to  each 
level  in  the  decision-making  hierarchy. 

4. 2   Recommended  Criteria  for  Extramural  Program  Management 

As  a  result  of  analysis  of  the  criteria  statements 
received  during  the  research  relevance  exercises  (Exercise  I 
and  II,  mentioned  above)  and  subsequent  discussions  with 

members  of  the  Institute's  Staff  and  Advisory  Council, 
Project  SOAP  recommended  the  following  four  criteria*  for 
evaluating  Extramural  research  projects  for  quality, 
importance,  and  cost: 


*For  a  more  comprehensive  explanation,  see  "Recommended 
Criteria  for  Extramural  Program  Management,"  Project  SOAP 
Notes  No.  17,  March  15,  1971. 
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I.   Quality  of  Research  ("Scientific  Merit") 

II.   Importance  (Relevance)  to  Health  Research  Programs 

III.   Distribution  of  Funds  Between  Programs  (Short- 
Term  Balance) 

IV.   Investment  in  Science  (Long-Range  Forecast) 

It  is  expected  that  these  "Guideline  Criteria"  will 

develop  for  the  Institute  a  more  explicit  pattern  of 

reasoning  for  guiding  the  decision-making  process  between 

the  Council,  the  Staff,  the  Study  Sections,  and  the  Division 

of  Research  Grants  of  NIH.   Later,  these  guidelines  can 

be  used  to  establish  a  basis  for  assessing  and  revising 

the  Institute's  Special  Emphasis  Research  Programs.   Thus, 

they  can  become  the  medium  for  communicating  Extramural 

Program  priorities  to  NIH,  the  Office  of  Management  and 

Budget,  the  Congress,  and  the  scientific  community 

at  large. 

4.3  Research  Investment  Procedure:   Dollar  Allocation 
by  Program  Area  (DAPAT 

After  exploring  several  possible  approaches  to  solving 

NIAID's  priority  management  problems,  Project  SOAP  concluded 

its  study  with  a  proposal  for  a  new  procedure.   The  "Dollar 

Allocation  by  Program  Area:   An  Investment  Procedure" 

was  presented  to  and  accepted  by  the  NIAID  Advisory 

Council  at  its  March  18,  1971,  meeting. 
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DAPA  is  an  alternative  to  decile  manipulation  as  a 
device  for  allocating  funds  to  Extramural  research  activi- 
ties.*  It  is  based  on  the  concept  of  capital  investment  to 
areas  of  socially  important  health  research,  within  the 
context  of  projects  supported  by  NIAID  Extramural  grants. 
The  procedure  is  designed  to  replace  the  present  "program 
relevance-decile  adjustment"  system,  while  also  preserving 
the  existing  practice  of  ordinal  ranking  for  "scientific 
merit"  by  the  Study  Sections. 

This  procedure  is  analogous  to  a  two-part  securities 
investment  portfolio,  such  as  that  for  common  stocks  and 
bonds.   Figure  4  (following  page)  shows  how  research  funds 
had  previously  been  allocated  and  served  to  indicate  to  the 
Advisory  Council  and  NIAID  staff  certain  funding  imbalances 
in  their  existing  program  structure.   This  illustrates  a 
feature  of  using  DAPA.   The  first  part  of  the  portfolio 
reflects  selected  investments  in  specified  Special  Emphasis 
Program  areas  (SERP's)  of  particular  importance  to  the 
mission  of  the  Institute.   The  purpose  of  these  SERP's 
is  to  increase  the  probability  of  yielding  "health  relevant" 
results  from  grant- supported  research.   The  second  part 
concerns  investments  of  Extramural  funds  in  support  of  free- 
ranging  investigator-inspired  research.   Grant  applications 

are  paid  in  order  of  peer-recommended  "scientific  merit" 
priority  scores. 


*For  a  detailed  description,  see  "DAPA  Research  Investment 
Procedure,"  Project  SOAP  Notes  No.  19,  June  15,  1971. 
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NIAID  Extramural  Research  Project  Grants 
DAPA  Research  Investment  Portfolio 
(March  1971) 


Program  Category 

Number  of 
Grants 

$  Million 
Amount 

Percent 
Extramural 
Budget 

I.  Special  Emphasis  Research  Programs 

A.  Clinical  Allergy 

58 

3.2 

6.3% 

B.  Transplantation  Immunology 

71 

3.7 

7.3% 

C.  Strep.  Disease  §  Sequelae 

25 

0.9 

1.8% 

D.  Drug  Resistance 

210 

7.1 

13.9% 

E.  Infectious  Hepatitis 

5 

0.3 

0.6% 

F.  Chronic  §  Degenerative  Diseases 

45 

1.5 

2.9% 

G.  Antiviral  Substances  (Interferon] 

116 

5.3 

10.4% 

H.  Biological  Regulation  of  Vectors 

40 

1.4 

2.8% 

I.   Immunity  to  Animal  Parasites 

44 

1.4 

2.8% 

SERP  Total 

614 

24.8 

48.8% 

II.  Investigator  Inspired 
Research  Projects 

565 

23.7 

46.5% 

III.  Other  Research  Projects 

66 

2.4 

4.7% 

TOTAL 

1,245 

$50.9 

100.0% 

Figure  4 
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4.3.1   The  PAPA  Procedure 

At  least  once  a  year,  the  Advisory  Council  meets  to 
advise  the  Institute  on  Extramural  Program  investment 
policy.   The  Institute  staff  provides  a  review  of  present 
SERP  activities,  with  summaries  of  the  distribution  of  funds 
to  grantees  in  each  support  area.   Staff  also  submits  a  pro- 
posed investment  plan  for  Council  approval,  which  includes 
necessary  historical  perspectives  of  specific  programs. . 
Recommendations  for  new  SERP's  or  terminations  of  present 
ones  would  require  support  documentation.   Incremental 
adjustments  within  previously  approved  guideline  ranges  need 

only  be  proposed  to  the  Council  for  approval. 

After  Staff's  presentation,  the  Council  may  deliberate 

the  merits  and  implications  of  the  plan  as  well  as  call  upon 
outside  experts  for  additional  information.   The  Council  then 
resolves  the  issue  by  "approving"  a  modification  of  the 
Institute's  investment  proposal.   This  includes  the  total 
dollars  and  percentage  range  to  be  allocated  for:   1)  free- 
ranging  research  projects,  and  2)  each  Special  Emphasis 
Research  Program  category.   Also  to  be  approved  (perhaps 
with  modifications)  are  statements  describing  each  SERP's 
area  (Program  Definition  Statements) ,  including  its  specific 
goals,  scope,  research  characteristics  and  a  brief  state-of- 
the-art  summary. 

Special  Subcommittees  of  the  Advisory  Council  ad  hoc 

advisors,  and  Institute  Staff  would  be  appointed  and 
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specifically  assigned  Extramural  program  planning  and 
evaluation  functions.   These  working  groups  would  meet 
during  periods  between  scheduled  Council  meetings  to  deter- 
mine the  nature  and  magnitude  of  each  approved  SERP  category, 
including  major  revisions  if  warranted.   Specialized  infor- 
mation and  competent  advice  from  other  advisory  groups 
(like  Study  Sections,  consultant  experts,  and  contract 
studies)  would  be  available  to  them. 

Pertinent  health  statistics  on  disease/disease  pro- 
blems, populations,  specific  disease  mortality  and  morbidity 
rates,  hospitalization  costs,  epidemiology,  and  related 
health  information  will  be  included.   The  use  of  these  data 
suitably  combined  in  an  appropriate  model  would  provide  a 
quantitative  basis  for  establishing  the  programmatic  "health 
relevance"  for  each  SERP  category. 

The  qualitative  judgments  of  the  Staff /Council  Sub- 
committee would,  of  course,  require  such  data  as  a  basis  for 
proposing  appropriate  levels  of  investment  and  establishing 
SERP  payment  priorities. 

In  brief,  the  DAPA  procedure  is  part  of  a  program 
management  discipline  for  distributing  limited  research 
funds.   It  provides  a  reasonable  method  for  Staff /Council 
interaction  in  balancing  budget  allocations  between  desired 
ends  (health  research  "needs")  and  available  means  (bio- 
medical research  opportunities) ,  while  at  the  same  time 
relieving  the  Advisory  Council  of  the  task  of  reviewing 
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each  individual  grant  proposal  recommended  by  the  Study- 
Sections  for  payment  approval.   (A  flow  chart  of  the 
DAPA  Procedure  is  shown  in  figure  5,  following  page.) 

Grants  supporting  free-ranging  (FR)  research 
investigators  would  continue  to  be  awarded  on  the  basis 
of  Study  Section  priority  scores.   Those  proposals 
which  further  qualify  for  support  under  one  of  the 
Institute's  Special  Emphasis  Research  Programs  would 
be  paid,  on  the  basis  of  similar  order  of  priority 
scores,  to  the  limit  of  designated  program  funds 
allocated  to  each  prior  approved  category. 

The  application  of  the  results  of  this  systematic 
planning  and  evaluating  process  would  become  the  responsi- 
bility of  each  SERP  Program  Manager  appointed  by  the 
Director,  subject  to  the  advice  and  continued  review  of 
his  subcommittee.   When  a  particular  SERP  category  has 
reached  maturity  (attained  that  "critical  mass"  of  funda- 
mental knowledge  required  for  applied  research- -as  the 
category  of  hepatitis  did  a  few  years  ago) ,  the  decision 
to  "transfer"  support  and  scientific  review  to  a  more 
appropriate  Intramural  or  Collaborative  research 
activity  within  the  Institute  could  be  made.   This 
ultimately  would  lead  to  more  direct  health  research 
applications  in  one  of  the  NIAID  Institute's  national 
clinical  centers. 


50 


3  s< 

o  9. 


m  id 


I 


(lEAOjddv) 


-\ 


All 

Pi 


f^~\  4->  o   ni  ■ 


0 


ma     \ 

oj    i/>        03 
■M  <n  o  W 

tO  <    4->  C/3 


L " 


<;  o  -h  o 
2  05  D-,  c/5 


Cu 

^ 

u 

n) 

3 

a 

<u 

u 

u. 

> 

a 

u 

u. 

u. 

< 

_J 

i_ 

* 

<k> 

Study 
Sections 
Review 
Ratings 

-p 
d 

x  e 

H    C 
•H 

m 
in 

< 

^ 

J 

k 

Xq.Turuiiiiioo  oijiausiDS  ipxessaa  iroxpauioTg 

I 


51 


Program  areas  and  their  budgets  would  be  reviewed 
regularly  by  Staff  and  modified  by  Council/Staff  agreement, 
as  new  and  more  current  information  is  obtained.   Progress 
or  stagnation  could  thereby  be  more  clearly  discerned,  with 
greater  assurance  of  the  Council's  making  timely  assessments 
as  well  as  fully-based  initial  decisions. 

Implicit  in  the  DAPA  approach  is  the  necessity  of 
professional  judgment  and  reliable  information  upon  which 
to  base  these  decisions.   Under  existing  conditions  of 
great  uncertainty  regarding  the  technological,  economic, 
and  social  feasibility  of  emphasizing  a  particular  field 
of  research,  this  procedure  provides  increased  flexibility 
for  guiding  the  Extramural  program  planning  and  evaluation 
activities.   Moreover,  it  offers  an  operational  method 
for  potentially  doing  a  better  job  with  the  skills 
and  resources  available. 

The  Project  SOAP  task  group  believes  that  the  use  of 
the  dollar  allocation  method  (DAPA)  could  become  an 
important  management  tool  for  science  administrators 
elsewhere  within  NIH  for  negotiating  the  "health  relevance" 
of  biomedical  research  programs  with  the  Office  of  Manage- 
ment and  Budget.   Further  testing  and  evaluation  of  the 
DAPA  Research  Investment  Procedure  was,  therefore, 
recommended.   As  noted  earlier,  initial  steps  have  been 
indicated  in  this  direction. 
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Because  of  its  potential  importance  for  long  range 
program  planning,  the  Task  Group  also  investigated  the  possi- 
bility of  developing  a  Biomedical  Research  Utility  Model  to 
aid  in  planning  and  evaluating  the  social/economic  impacts 
of  major  health  research  programs.   This  part  of  the  study, 
which  is  contained  in  Project  SOAP  Notes  No.  14,  was  not 
implemented  due  to  initial  reluctance  on  the  part  of  NIAID 
Staff  to  embrace  so  sophisticated  a  modeling  method  without 
a  greater  understanding  of  and  confidence  in  the  validity 
of  its  results.   The  model  basically  employed  statistical 
data  from  epidemiological  sources  to  evaluate  the  National 
importance  of  a  particular  disease. 

It  attempted  to  estimate  the  "economic  cost"  to  the 
Nation  in  terms  of  medical  expenses,  hospitalization  costs, 
lost  work  time,  and  a  factor  for  human  discomfort.   The 
approach  is  similar  to  that  for  estimating  insurance  risk, 
but  does  not  have  the  advantage  of  clear  empirical  data. 
For  this  reason,  it  was  felt  the  actual  application  of  a 
Biomedical  Research  Utility  Model  was  premature. 

(Developing  such  models  is,  of  course,  a  difficult 
and  demanding  task,  but  an  inescapable  one  in  arriving 
at  an  appropriate  quantitative  basis  for  the  inherently 
quantitative  decisions  of  resource  allocation.) 
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5.   CONCLUSION 

Summing  up:   this  case  study  documents  the  methodology 
and  substantive  information  developed  by  the  Project  SOAP 
task  group  during  its  two-year  assignment  to  perform  analyti- 
cal studies  for  the  National  Institute  of  Allergy  and  Infec- 
tious Diseases  of  the  National  Institutes  of  Health  between 
1969  and  1971. 

Project  SOAP  created  for  NIAID  a  clearer  understanding 
and  a  more  focused  awareness  of  the  nature  of  its  resource 
allocation  problems,  particularly  those  of  the  Extramural 
Grant  Program.   This  first  step  was  reflected  in  the  action 
taken  by  the  NIAID  Advisory  Council  at  its  meeting,  June  21, 
1971.   (See  following  page,  which  was  abstracted  from  the 
official  minutes  of  the  June  1971  Council.) 

It  would  be  unreasonable  to  expect  that  any  task  group 
could  attain  so  profound  an  insight  into  the  myriad  problems 
of  a  complex  organization  that  it  could  actually  identify  any 
one  problem  as  the  crux  of  matter  and  determine  how  it  should 
be  solved.   There  are  certain  problems  far  too  complicated 
for  systems  analysis  as  yet  to  comprehend.   Nevertheless, 
given  its  limited  scope  of  understanding  in  approaching  the 
management  situation  it  faced,  the  Project  SOAP  task  group 
stimulated  NIAID  to  perceive  its  new  role  in  a  changing 
environment:  that  of  managing  its  resources  explicitly. 
With  this  new  awareness,  NIAID  was  then  in  a  far  better  posi- 
tion to  accept  and  adopt  the  use  of  recommended  management 

methods  for  improving  its  decision-making  processes. 
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Report  of  NIAID  Advisory  Council  Meeting 

(Abstracted  from  Official  Minutes  of  the  June  1971  Council 
National  Institute  of  Allergy  and  Infectious  Diseases/NIH) 

Allocation  of  Extramural  Resources 
(.Informational  J 

The  first  step  in  implementation  of  the 
"Dollar  Allocation  Program"  is  taken. 

In  taking  the  first  step  in  the  implementation  of  the 
"Dollar  Allocation  Program"  which  has  evolved  from  the 
Institute-sponsored  SOAP  study,  Council  reviewed  the  Special 
Emphasis  programs  of  the  Institute,  including  the  definitions 
of  these  programs,  and  the  funds  which  might  be  allocated  to 
them.   Rather  than  following  the  past  procedure  of  raising 
selected  research  grant  applications  to  payable  priority 
scores  they  concluded  that  during  FY  72,  funds  should  be 
allocated  to  specific  programs  to  enable  the  payment  of  more 
applications  assigned  to  these  programs.   The  Council  would 
review  this  allocation  at  each  of  its  subsequent  meetings  and 
would  not,  with  the  present  limit  of  funds,  approve  the  pay- 
ment of  research  grants  beyond  the  upper  60%  of  approved 
applications  in  any  of  the  programs  receiving  special 
allocation. 

In  the  future,  Council  will  be  emphasizing  review  of  programs 
rather  than  review  of  individual  projects  within  these  pro- 
grams.  The  Council  chose  to  continue  the  present  discipline- 
oriented  subcommittee  system  for  the  review  of  Institute 
programs  as  well  as  for  the  grants  assigned  to  them. 

Background: 

In  1969  an  Interagency  Agreement  between  the  Technical 
Analysis  Division  of  the  National  Bureau  of  Standards  and  the 
NIAID  was  entered  into  for  the  purpose  of  developing  criteria 
for  the  selection  of  grants  and/or  programs  for  preferential 
treatment  to  better  meet  the  needs  of  the  health  problems  of 
the  country.   The  Criteria  Subcommittee  of  Council  was 
appointed  to  act  as  liaison  between  Project  SOAP  (Systems/ 
Operations  Analysis  of  Programs) ,  the  working  group  of  this 
agreement,  and  the  Council  in  these  efforts.   After  tenta- 
tively exploring  a  number  of  possible  avenues,  the  Dollar 
Allocation  for  Program  Areas  was  presented  to  and  accepted 
by  Council  at  the  March  1971  meeting. 

Council  instructed  staff  to  redefine,  in  a  more  concise 
manner,  the  Special  Emphasis  areas  of  the  Institute  for  con- 
sideration by  Council  at  the  June  meeting.   It  was  agreed 
that  the  dollar  allocation  approach  would  be  entered  into 
gradually.   Program  definitions  were  reviewed  carefully, 
revised  by  staff  and  were  presented  to  this  Council  for  con- 
sideration. 
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PROJECT  SOAP  ISSUES  PRELIMINARY  FINDINGS  ON 
'HEALTH  RELEVANCE'  RATING  EXERCISE 


A  recent  interagency  agreement  between  IAT's  Technical  Analysis  Division 
(TAD)  and  the  National  Institute  of  Allergy  and  Infectious  Diseases  (NIAID)  of  the 
National  Institutes  of  Health,  resulted  in  the  creation  of  Project  SOAP.  Not  a  de- 
tergent, Project  SOAP  (System/Operations  Analysis  Programs)  is  concerned  with 
developing  quantitative  methods  for  determining  priorities  in  allocating  NIAID' s 
limited  research  funds.  One  issue  considered  was  whether  it  would  be  feasible  to  give 
some  weight  to  the  "health  relevance"  of  research  projects  in  deciding  whether  to 
fund  extramural  research  grant  applications.  Preliminary  findings  by  TAD  indicate 
that  it  is  indeed  practical  and  possible  to  consider  the  health  relevance  of  a  pro- 
posed research  project  as  well  as  its  "scientific  merit"  implications.  Implicit  in 
these  findings  is  the  need  for  development  of  "criteria"  for  health  relevance,  and 
work  is  currently  in  progress  along  those  lines. 

Above,  far  right,  Dr.  W.  E.  Cushen,  Chief  of  TAD,  presents  the  results  of  the 
health  relevance  analysis  to  (left)  Dr.  Dorland  J.  Davis,  Director  of  NIAID,  and  Dr. 
C.  West  Churchman,  Professor  of  Business  Administration,  University  of  California, 
and  Chairman  of  the  NIAID  Advisory  Council  Subcommittee  on  Criteria. 


Reprinted  from  the  NBS  Standard,  Vol.  XVI,  No.  2  (February  1971) 
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APPENDIX 


(Sections  A  and  B) 


SECTION  A  -  LIST  OF  NOTES  AND  WORKING  PAPERS 


PROJECT  SOAP  NOTES* 

Notes 
No.       Author 

Title 

Date 

1.  Robert  S.  Cutler 

"Definition  of  Terms" 

Aug.  29,  1969 

2.  Robert  S.  Cutler 

"Systems  Concepts  Re: 
Program  Management" 

Sept.  26,  1969 

3.  Vincent  A.  Martino   "Systems  Description  of        Dec.  15,  1969 
Robert  S.  Cutler    Collaborative  Research  Program" 

4.  Robert  S.  Cutler    "Extramural  Program:  Research    Jan.  30,  1970 
Vincent  A.  Martino   Grants  Award  System  Description" 

5.  Project  SOAP       "Task  Group  Briefing  to  NIH"     Feb.  5,  1970 

6.  Dr.  Julius  Lieblein  "Preliminary  Analysis  of  Data,   Mar.  15,  1970 

NIAID  Decile  System" 

7.  Edward  G.  Neigut    "Prospective  Assistance  Areas    Mar.  31,  1970 

for  NIAID  Collaborative  Programs" 

8.  Robert  S.  Cutler    "Criteria  Subcommittee  Meeting,"  May  29,  1970 

Berkeley,  California, 
April  25,  1970 

9.  Vincent  A.  Martino   "Analysis  of  Issue:  Relevance    June  30,  1970 

Criteria" 

10.  Vincent  A.  Martino   "Preliminary  Study  of  Group     Aug.  30.  1970 

Assessment  of  Health  Relevance, 
Exercise  I" 

11.  Harold  R.  Millie    "In-Basket  Survey  of  Public      Oct.  1,  1970 

Inquiries  Regarding  NIAID 
Health  Information" 

12.  Dr.  Julius  Lieblein  "Review  of  Report:  Change  in    Oct.  29,  1970 

Method  of  Calculation  of  Deciles" 

13.  Robert  S.  Cutler    "Unbundling  Criteria:  Science    Dec.  4,  1970 

Merit/Health  Relevance" 

14.  Dr.  Emmett  Keeler    "Models  of  Disease  Costs  and    December  1970 

Their  Use  in  Medical  Research 
Resource  Allocations" 


*Copies  of  Notes  are  available  from:  Project  SOAP,  Technical  Analysis 
Division,  National  Bureau  of  Standards,  Washington,  D.C.  20234 

A-l 


Notes 
No. 


Author 


Title 


Date 


15.  Stephen  P.  Strickland 


16.  Robert  S.  Cutler 
Thomas  A.  Goldman 


17.  Robert  S.  Cutler 
Vincent  A.  Martino 

18.  Robert  S.  Cutler 

19.  Robert  S.  Cutler 
Vincent  A.  Martino 

20.  Robert  S.  Cutler 


"Policy  Making  in  Biomedical    Mar.  15,  1971 

Research:  A  Perspective  on 

Relevance" 


"Thoughts  from  Exercise  II 
Pertaining  to  Social/Economic 
Impact  Criteria" 

"Recommended  Criteria  for 
Extramural  Program  Management" 


Mar.  15,  1971 


Mar.  15,  1971 


"Recommendations  for  Council"   Mar.  18,  1971 

"DAPA  Research  Investment       June  1,  1971 
Procedure" 

Exercise  II  Report:  "Biomedical  June  30,  1971 
Research  Relevance  Criteria" 
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WORKING  PAPERS 


Historical  Perspective  for  Relevance  Rating  Oct.  13,  1969 

(P.  Reeder) 
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